Make tests work on hardened runner by pietern · Pull Request #4875 · databricks/cli

pietern · 2026-03-31T14:47:08Z

Summary

Configure JFrog Artifactory as Go module proxy via OIDC for Linux and Windows runners
Disable run-local-node acceptance test (needs npm registry access via JFrog)
Add setup-build-environment to testmask triggers so CI changes test all targets
Switch secondary test jobs (aitools, ssh, pipelines) to use custom runner groups

Test plan

Verify make test passes on Linux, Windows, macOS
Verify test-exp-aitools, test-exp-ssh, test-pipelines are triggered and pass

This pull request was AI-assisted by Isaac.

Co-authored-by: Isaac

The hardened runner blocks direct access to public registries. Use JFrog Artifactory as a proxy with OIDC authentication. Co-authored-by: Isaac

The jf goc command only works with jf go, not native go commands. Set GOPROXY and .netrc for native go toolchain. Co-authored-by: Isaac

Replace jf config show (no JSON format flag) with direct OIDC token exchange, matching the pattern from the hardened runner docs for uv/cargo. Co-authored-by: Isaac

Use jf goc + jf go mod download to pre-populate the module cache via JFrog. Native go commands then work from cache. Co-authored-by: Isaac

Co-authored-by: Isaac

eng-dev-ecosystem-bot · 2026-03-31T15:13:44Z

Commit: 5270413

Run: 23807052984

	Env	🟨KNOWN	🔄flaky	💚RECOVERED	🙈SKIP	✅pass	🙈skip	Time
🟨	aws linux	7			10	270	810	7:21
🟨	aws windows	7			10	272	808	6:46
💚	aws-ucws linux			7	10	366	726	7:58
💚	aws-ucws windows			7	10	368	724	6:33
💚	azure linux			1	12	273	808	6:36
💚	azure windows			1	12	275	806	6:24
🔄	azure-ucws linux		2	1	12	369	722	7:22
💚	azure-ucws windows			1	12	373	720	6:10
💚	gcp linux			1	12	269	811	5:30
💚	gcp windows			1	12	271	809	6:27

19 interesting tests: 10 SKIP, 7 KNOWN, 2 flaky

	Test Name	aws linux	aws windows	aws-ucws linux	aws-ucws windows	azure linux	azure windows	azure-ucws linux	azure-ucws windows	gcp linux	gcp windows
🟨	TestAccept	🟨K	🟨K	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R
🙈	TestAccept/bundle/resources/permissions	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions	🟨K	🟨K	💚R	💚R	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct	🟨K	🟨K	💚R	💚R
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform	🟨K	🟨K	💚R	💚R
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions	🟨K	🟨K	💚R	💚R	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct	🟨K	🟨K	💚R	💚R
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform	🟨K	🟨K	💚R	💚R
🙈	TestAccept/bundle/resources/postgres_branches/basic	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_branches/recreate	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_branches/update_protected	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_branches/without_branch_id	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_endpoints/basic	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_endpoints/recreate	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_projects/update_display_name	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/synced_database_tables/basic	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/ssh/connection	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🔄	TestFilerWorkspaceNotebook	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p	✅p	✅p
🔄	TestFilerWorkspaceNotebook/scalaNb.scala	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p	✅p	✅p

Top 21 slowest tests (at least 2 minutes):

duration	env	testname
3:44	gcp windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:41	gcp linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:39	gcp windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:31	aws linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:23	gcp linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:20	azure-ucws linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:11	aws linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:11	aws-ucws linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:08	azure windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:05	aws windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:58	aws windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:52	azure-ucws windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:45	azure linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:44	azure windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:42	azure-ucws linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:42	aws-ucws linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:40	aws-ucws windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:40	aws-ucws windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:27	azure windows	TestAccept
2:20	azure linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:15	azure-ucws windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct

Co-authored-by: Isaac

The hardened runner doesn't have Node.js pre-installed, which is needed by the run-local-node acceptance test. Co-authored-by: Isaac

The test requires npm registry access which is blocked on the hardened runner. Disable until npm is routed through JFrog. Co-authored-by: Isaac

Co-authored-by: Isaac

The existing runners are already hardened, so no need to switch to the hardened-optin group. Co-authored-by: Isaac

Windows runners are also hardened and can't reach proxy.golang.org. Co-authored-by: Isaac

PowerShell splits -modfile=tools/go.mod into separate tokens. Co-authored-by: Isaac

This ensures all test jobs that use the shared action get the JFrog Go module proxy configured on hardened runners. Co-authored-by: Isaac

…test jobs to custom runners Co-authored-by: Isaac

Co-authored-by: Isaac

github-actions · 2026-04-01T09:41:09Z

Suggested reviewers

Based on git history of the changed files, these people are best suited to review:

@denik -- recent work in .github/workflows/, .github/actions/setup-build-environment/, acceptance/cmd/workspace/apps/run-local-node/
@fjakobs -- recent work in tools/testmask/, .github/workflows/

Confidence: high

Eligible reviewers

Based on CODEOWNERS, these people or teams could also review:

@andrewnester, @anton-107, @shreyas-goenka, @simonfaltum

_{Suggestions based on git history of 6 changed files (5 scored). See CODEOWNERS for path-specific ownership rules.}

denik · 2026-04-01T10:57:35Z

acceptance/cmd/workspace/apps/run-local-node/test.toml

@@ -1,5 +1,6 @@
+Badness = "need to enable NPM registry access"


We should also consider vendoring dependencies here like we did with Python.

Evaluate hardened runner for CI

4007149

Co-authored-by: Isaac

pietern temporarily deployed to test-trigger-is March 31, 2026 14:47 — with GitHub Actions Inactive

Fix runner group name

62449de

Co-authored-by: Isaac

pietern temporarily deployed to test-trigger-is March 31, 2026 14:49 — with GitHub Actions Inactive

Configure JFrog proxy for Go on hardened runner

cc5cd36

The hardened runner blocks direct access to public registries. Use JFrog Artifactory as a proxy with OIDC authentication. Co-authored-by: Isaac

pietern temporarily deployed to test-trigger-is March 31, 2026 14:55 — with GitHub Actions Inactive

Set GOPROXY and auth for JFrog Go proxy

61ec386

The jf goc command only works with jf go, not native go commands. Set GOPROXY and .netrc for native go toolchain. Co-authored-by: Isaac

pietern temporarily deployed to test-trigger-is March 31, 2026 14:59 — with GitHub Actions Inactive

Use manual OIDC token exchange for Go proxy auth

053cc50

Replace jf config show (no JSON format flag) with direct OIDC token exchange, matching the pattern from the hardened runner docs for uv/cargo. Co-authored-by: Isaac

pietern temporarily deployed to test-trigger-is March 31, 2026 15:02 — with GitHub Actions Inactive

Use JFrog CLI directly for Go module downloads

0b8b171

Use jf goc + jf go mod download to pre-populate the module cache via JFrog. Native go commands then work from cache. Co-authored-by: Isaac

pietern temporarily deployed to test-trigger-is March 31, 2026 15:04 — with GitHub Actions Inactive

Empty commit to test hardened runner queue time

3a06528

Co-authored-by: Isaac

pietern temporarily deployed to test-trigger-is March 31, 2026 15:13 — with GitHub Actions Inactive

Revert Makefile to run full test suite

2d5fc49

Co-authored-by: Isaac

pietern temporarily deployed to test-trigger-is March 31, 2026 15:18 — with GitHub Actions Inactive

Add setup-node to build environment

5270413

The hardened runner doesn't have Node.js pre-installed, which is needed by the run-local-node acceptance test. Co-authored-by: Isaac

pietern temporarily deployed to test-trigger-is March 31, 2026 16:01 — with GitHub Actions Inactive

Disable run-local-node test on hardened runner

c9b6413

The test requires npm registry access which is blocked on the hardened runner. Disable until npm is routed through JFrog. Co-authored-by: Isaac

pietern had a problem deploying to test-trigger-is April 1, 2026 08:35 — with GitHub Actions Failure

Remove setup-node; run-local-node is the only test needing it

9831754

Co-authored-by: Isaac

pietern had a problem deploying to test-trigger-is April 1, 2026 08:36 — with GitHub Actions Failure

pietern changed the title ~~Evaluate hardened runner~~ Make tests work on hardened runner Apr 1, 2026

Revert runner group to existing large runners

bccbb84

The existing runners are already hardened, so no need to switch to the hardened-optin group. Co-authored-by: Isaac

pietern had a problem deploying to test-trigger-is April 1, 2026 08:40 — with GitHub Actions Failure

Enable JFrog proxy for Windows runners too

7363747

Windows runners are also hardened and can't reach proxy.golang.org. Co-authored-by: Isaac

pietern had a problem deploying to test-trigger-is April 1, 2026 08:47 — with GitHub Actions Failure

denik approved these changes Apr 1, 2026

View reviewed changes

Use bash shell for JFrog module download step

9fca8f3

PowerShell splits -modfile=tools/go.mod into separate tokens. Co-authored-by: Isaac

pietern had a problem deploying to test-trigger-is April 1, 2026 08:51 — with GitHub Actions Failure

Move JFrog proxy setup to shared setup-build-environment action

eacae04

This ensures all test jobs that use the shared action get the JFrog Go module proxy configured on hardened runners. Co-authored-by: Isaac

pietern had a problem deploying to test-trigger-is April 1, 2026 09:14 — with GitHub Actions Failure

Add setup-build-environment as testmask trigger and switch secondary …

c0cc672

…test jobs to custom runners Co-authored-by: Isaac

pietern had a problem deploying to test-trigger-is April 1, 2026 09:20 — with GitHub Actions Failure

Fix YAML formatting for commented-out Windows matrix entry

789e563

Co-authored-by: Isaac

pietern had a problem deploying to test-trigger-is April 1, 2026 09:28 — with GitHub Actions Failure

andrewnester approved these changes Apr 1, 2026

View reviewed changes

pietern marked this pull request as ready for review April 1, 2026 09:40

pietern requested review from anton-107, shreyas-goenka and simonfaltum as code owners April 1, 2026 09:40

pietern merged commit bb8b828 into main Apr 1, 2026
24 of 25 checks passed

pietern deleted the eval-hardened-runner branch April 1, 2026 09:50

denik reviewed Apr 1, 2026

View reviewed changes

mihaimitrea-db mentioned this pull request Apr 1, 2026

Configure JFrog Artifactory as PyPI proxy for hardened runners databricks/databricks-sdk-py#1379

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make tests work on hardened runner#4875

Make tests work on hardened runner#4875
pietern merged 17 commits intomainfrom
eval-hardened-runner

pietern commented Mar 31, 2026 •

edited

Loading

Uh oh!

eng-dev-ecosystem-bot commented Mar 31, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Apr 1, 2026

Uh oh!

Uh oh!

denik Apr 1, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		@@ -1,5 +1,6 @@
		Badness = "need to enable NPM registry access"

Conversation

pietern commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

eng-dev-ecosystem-bot commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 1, 2026

Suggested reviewers

Eligible reviewers

Uh oh!

Uh oh!

denik Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pietern commented Mar 31, 2026 •

edited

Loading

eng-dev-ecosystem-bot commented Mar 31, 2026 •

edited

Loading

denik Apr 1, 2026 •

edited

Loading